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Background 


At the request of its 194 Member States in May 2015, the World Health Organization (WHO) 
convened a broad coalition of experts to develop an R&D Blueprint for Action to Prevent Epidemics. 
The R&D Blueprint presents options to reduce the time lag between the identification of a nascent 
outbreak and approval of the most advanced products that can be used to save lives and stop larger 
crises. It focuses on severe emerging diseases with potential to generate a public health emergency, 
and for which no, or insufficient, preventive and curative solutions exist. 


Activities under the R&D Blueprint are organized into three clusters of activities. The second cluster 
focuses on accelerating research and development processes. It includes: 


1. Assessing epidemic threat and defining priority pathogens; 


2. Developing R&D roadmaps to accelerate evaluation of diagnostics, therapeutics and vaccines; 
and 


3. Outlining appropriate regulatory and ethical pathways. 


This methodology is intended to help identify the top global disease threats as part of an ongoing 
process to reassess priorities in light of changing circumstances. 


Evolution of the methodology 


This methodology was originally outlined by participants at the WHO Consultation for Prioritization of 
Pathogens held in Geneva, Switzerland from 8-9 December 2015." Experts at that meeting reviewed 
best practice in conducting disease prioritization processes, amending it to fit the needs of the R&D 
Blueprint (Annex 1). They also reviewed practical examples of national disease prioritization 
processes and benefit from the input of experts involved in such assessments. There was broad 
agreement that any methodology for prioritizing diseases and pathogens would need to be 
transparent and be responsive to changes in understanding and current events. 


It was recommended that two separate prioritization processes were necessary: 


1. An annual prioritization exercise to review and revise a list of prioritized diseases and 
pathogens (Part 1); and 


2. A separate process for dealing with a new disease or pathogen, or one that is presenting in a 
new manner and likely to cause a public health emergency (Part 2). 


There was agreement that the methodology would need regular review, at least every three years, to 
ensure that it continues to offer the optimum approach for prioritizing diseases. Such a review of the 
methodology might be held adjacent to, but separate from, the annual prioritization exercise. 


Based upon this guidance, the methodology was further developed by WHO through the development 
of a number of disease scenarios (Annex 2) and a decision tree for new diseases. The R&D 
Blueprint’s Scientific Advisory Group reviewed drafts of both parts of the methodology. 


The entire methodology was reviewed by a cross-discipline expert group at an informal consultation 
convened in Geneva, Switzerland from 17-18 November 2016. The meeting brought together experts 
in human and animal health, epidemiology, applied mathematics and safety as well as relevant 
researchers and clinicians. It validated a general approach, endorsing a system of annual reviews, 
biennial methodology reviews, supplemented as necessary with emergency reviews. The annual 
reviews will utilize a combination of rounds of the Delphi technique, questionnaires and multi-criteria 
decision analysis to review and update the R&D Blueprint's priority list of diseases. The meeting also 
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examined a set of draft tools to be used in this process, refining some and detailing a roadmap for the 
completion of others. 


This methodology was the result of the comments, feedback, and improvements received during and 
after the November 2016 methodology review. It was validated through a silence procedure in 
January 2017. It is anticipated that it will be reviewed again before the end of 2019. 


The need for a different prioritization approach 


There have been numerous past efforts to identify a subset of infectious diseases that needs to be 
prioritized for research, development, preparedness or other pre-emptive action. A recent systematic 
review of many such processes noted different methodological approaches including bibliometric 
indexes, Delphi techniques, Multi-Criteria Decision Analysis (MCDA), qualitative algorithms, and 
questionnaires. Some were more suitable than others for the prioritization under the R&D Blueprint 


(Table 1). 


Scenarios Methodology Comments 
ae E Both methods are suitable for ranking a large volume of pathogens 
: within a short time period or with limited resources. 
of pathogens algorithm 
Scoping exercise to H-index or As both methods can quickly rank a large volume of pathogens, they 
generate an initial qualitative can be used to provide a short list for risk ranking using a more 
ranking for further study |algorithm comprehensive technique. 


Comprehensive risk 
ranking including novel, 
emerging and 
established infections 


MCDA or Delphi 


Both methods provide a comprehensive method for risk ranking. 
Where resource is restricted, consider limiting the number of criteria 
or the number of diseases for ranking. 


Emerging infections with 


In lieu of standard data, such as burden of disease, h-index can 


little published data H-index indicate a level of professional interest/concern which may be used 
about them as an informal proxy measure of disease impact. 
This method combines expert opinion and evidence (where 
Qualitative available). The qualitative nature allows for greater flexibility in 
algorithm decision-making and for the detailed recording of that rationale. This 
9 is particularly useful in emerging infections where decisions may be 
more based on expert opinion than epidemiological data. 
Qualitative In qualitative methodologies, including a mechanism for respondents 
algorithm or to identify gaps in knowledge or areas for further work could lead to 


questionnaires 


improved evidence upon which to base future decisions. 


MCDA 


This method can incorporate information from a variety of sources, 
which is useful in emerging infections where information is sparse. 
Ranking the risk of alternative scenarios is suitable for situations 
where there is less certainty about the potential course of the 
disease. Additionally, new information can be incorporated as it 
emerges, without needing to rerun the entire ranking exercise 


TABLE 1: Scenarios and suggested methodologies for risk-ranking exercises” 


The current WHO prioritization process is most closely aligned to a “Comprehensive risk ranking 
including novel, emerging and established infections”. As a result, a combination of Delphi techniques 
and MCDA was recommended. The methodology also makes use of the questionnaires, which were 
highlighted as useful in cases where there may be little published data available on the diseases 
involved. As a result, this methodology uses a Delphi approach to a short list of diseases to review in 
more detail, and MCDA to assess those diseases, questionnaires to illicit data from participants and a 
final round of the Delphi technique to review the provisional results. 


The review of best practice in disease prioritization also highlighted a number of ways in which past 
WHO processes might usefully be improved: 


? Table from: O’Brien, E. C., et al. "Best practices in ranking communicable disease threats: a literature review, 2015." Euro 
surveillance: bulletin Européen sur les maladies transmissibles= European communicable disease bulletin 21.17 (2016). 


http://www.eurosurveillance.org/ViewArticle.aspx?Articleld=22459 
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Reporting lacked detail, as it was a report of a meeting to give participants experience of 
such an exercise. Unclear how criteria were developed. Potential sources of bias and 
mitigations are not reported. The publication was not peer-reviewed and it is unclear if 
any other review took place. Implementation issues were not discussed but Delphi 
scoring was limited to one round. Did not meet all of the key communicable disease 
facets. 95% confidence intervals used to aid discussion of discrepancies in scoring. 


As a result, a single round of the Delphi technique has been doubled for the initial triaging of diseases 
and supplemented with a more rigorous MCDA assessment. 


Rather than making use of a single meeting the process by which the prioritization methodology is 
developed has been strengthened. First, a general framework was agreed by a group of eminent 
experts during the 2015 prioritization meeting. Details criteria for assessment were then added. The 
resulting draft methodology was presented to the Scientific Advisory Board of the R&D Blueprint and 
amended in light of their recommendations. The implementation of the modelling used in this 
methodology was reviewed by leading international experts and revised following their feedback. The 
revised methodology was presented to, reviewed, and further developed by, an additional group of 
external experts, including experts in public health, infectious disease, modelling of disease and 
relevant methodologies. Finally, the methodology itself is being made public. 


Two particular recommendations by the WHO R&D Blueprint Scientific Advisory Board have 
particularly shaped this document: (1) the methodology should be better documented and made 
publicly available; and (2) to develop estimates of confidence in the results achieved: 


e On documentation - the reports of the initial prioritization exercise in 2015, the 
methodological review in 2016 and the annual review of the priority list in 2017 all document 
the evolution of this methodology and its implementation. These reports are publicly available 
and published on WHO’s website. This document records how WHO creates its priority list 
and is to be made publicly available. It will be supplemented by a peer-reviewed article 
following its use to revise the list of priority diseases. The entire methodology will be reviewed 
through a dedicated, transparent process every few years. 


e Onestimates of confidence - to better structure the organization and analysis of the decisions 
being taken an Analytic Hierarchy Process (AHP) was employed. This technique, based upon 
developments in mathematics and psychology, is a framework for structuring decision making 
- representing and quantifying relevant elements, connecting them to overall goals, and for 
evaluating alternative solutions. It also enables the consistency of different views amongst 
participant to be analysed. The consistency indicates the level of randomness in the 
comparison. The application of the AHP approach in this context and for the specific aims of 
the prioritization process was reviewed by a number of prominent academics in the field. 
Furthermore, the methodology makes use of error propagation techniques to produce 
uncertainty estimation — effectively using the differences between opinions to assess 
discordance on the ranking. 


Particular attention was placed on identifying and mitigating sources of bias. In 2010, Pannucci and 
Wilkins identified a number of different biases present in clinical trials. A number of similar biases 
could be present in this prioritization exercise. Tailored measures have been taken to minimize their 
possible impact: 


e Flaws in study design — there is a need to: (a) clearly define risk and outcome, preferably with 
objective or validated methods; and (b) standardize and blind data collection. This 
methodology is effectively a protocol validated by the relevant expert community. It makes 


3 European Centre for Disease Prevention and Control. Best practices in ranking emerginginfectious disease threats. 
Stockholm: ECDC; 2015. http://ecdc.europa.eu/en/publications/Publications/emerging-infectious-disease-threats-best- 
practices-ranking.pdf 

“C.J. Pannucci, & E.G. Wilkins. Identifying and Avoiding Bias in Research, Plastic Reconstructive Surgery, August 2010; 
126(2): 619-625. https://www.ncbi.nim.nih.gov/pmc/articles/PMG2917255/pdf/nihms-198809.paf 


3 


use blinded surveys and questionnaires for much of its data collection, specifically to ensure a 
standardized, blinded approach; 


Selection bias — Subjects of the study should be selected through a rigorous criteria designed 
to avoid confounding results. The standardized approach to identifying a long list of diseases 
to be considered, a transparent triaging process and the use of surveys is al intended to 
provide a rigorous approach to selecting the disease and pathogens selected for inclusion 
and prioritization. Furthermore, the identification of key knowledge sets, expertise and a broad 
geographic representation of experts all contribute to minimizing the impact of selection bias 
on participants; 


Interviewer bias — Standardizing interviewer's (or participant’s) interaction with the process is 
important. This methodology uses surveys during the MCDA process to ensure a 
standardized experience; 


Chronology bias — given the comparative infrequency of public health emergencies, this 
methodology will, by necessity, make use of historical data. Adopting approaches allowing 
assessment of confidence in the results produces help to mitigate such a bias. 


Recall bias — another major challenge for prioritization processes is that those participating 
will be privy to, and drawing upon, knowledge and data that might not be in the public domain. 
This makes it difficult to ensure arguments being made are supported with objective data 
sources. The use of Delphi techniques to share knowledge amongst participants and enable 
a limited peer review during discussion of the diseases and pathogens covered, helps explore 
the technical basis of data not in the public domain. Efforts to approximate confidence levels 
in the results achieved, may also help to alleviate this bias. 


PART 1 
Annual Prioritization Exercise 


A. Overview of the annual prioritization exercise 


This annual prioritization exercise is the final step in a larger process. It has been preceded by efforts 
to identify criteria against which to assess diseases and pathogens, the weighting of the criteria 
according to their relative importance (Annex 1), and a thorough review of the methodology. 


In general, the annual exercise is intended to move through a number of sequential steps to get from 
all known diseases and pathogens to a short-list of around 10 pathogens to be prioritized under the 
R&D Blueprint (Figure 1). 


e Landscape analysis e Ranking diseases by e Promoting the 


e Delphi process MCDA algorithm prioritized list 


e Assessing confidence 
in the prioritized list 

e Reviewing the results 
(Delphi process) 


Triaging list of diseases Ranking the long list of Post-prioritization 
to be considered diseases & pathogens activities 


Figure 1: Overview of the annual prioritization exercise 


As a first phase, it is necessary to have prepared a sub-set of diseases and pathogens that will be 
considered for potential prioritization. Early in the annual review process it will be important to 
understand whether there have been any technical developments relevant to diseases currently 
prioritized (Section B). This will be fed into subsequent discussions. In any given annual review, it will 
not be possible to score and rank every known disease or pathogen. A triaging process is needed 
(Sections D and E), drawing upon a range of potentially relevant disease scenarios (Annex 2). 


A second phase is then to bring together a wide range of expertise (Section C). These experts will 
review candidate diseases and pathogens. They will score them according to the predetermined 
criteria (Section F), in line with the identified weightings (Section G), using established procedures 
(Sections H-K). 


Successful completion of this exercise will result in a revised list of priority diseases and pathogens. 
An assessment as to the degree of confidence in the results produced may then be necessary 
(Section L). A subsequent round of discussion to reflect on the process might also be necessary 
(Section M). 


A final phase will then promote the results of the annual prioritization exercise (Section N). This might 
include WHO press releases, the development of a publication, or other tools as required. 


Setting aside the initial planning activities to be undertaken by WHO and their commissioning of a 
landscape analysis, an annual review exercise will take 3-4 months to conduct (Figure 2). There are 
steps that will need to be taken in advance of the annual review meeting, such as setting up the 
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Prioritization Committee (Sections C) and triaging the diseases to be reviewed (Sections D). Other 
activities will form part of the meeting itself, such as reviewing the long list of disease and pathogens 
(Section E), assessing them against the weighted criteria (Section H), compiling a priority list (Section 
K), assessing confidence in it (Section L) and reflecting on the process (Section M). A few, such as 
promoting the results (Section N) occur after the meeting. 


Convene Prioritization Compile draft long list of diseases, distribute Beview resulte: réscore 8 
Committee & identify methodology & landscape analysis, & reflect on rocess 
additional participants identify experts to introduce diseases P 

Participants to review old list & Discuss diseases & use survey More detailed publication 

recommend additions 
Review long Short summary of 
Issue Invites list & triage findings 
2 months 6 weeks 1 month 2 weeks Startof Middle of End of 48hrs 1 month 

prior prior prior prior review review review after after 


Figure 2: Timings of key steps within an annual prioritization exercise 


B. Landscape analysis 


Prior to the annual prioritization exercise, WHO should commission a review of technical 
developments associated with the diseases and pathogens on the most recent priority list. This could 
usefully cover both developments in understanding, such as those generated by basic research, as 
well as those in more applied fields such as progress in developing the necessary medical 
countermeasures. The aim of the landscape analysis is to identify developments which might lead 
participants to think very differently about the relative risks posed by the disease or pathogen in the 
context of the R&D Blueprint. It is envisaged that such major developments are comparatively rare. 


The results of this landscape analysis will be fed into the annual exercise, preferable through a 
presentation near the start of the meeting (Annex 3). It will provide important background information 
for participant and help to ensure common ground for subsequent discussions. 


This landscape analysis will need to be carried out sufficiently early in the annual cycle so as to allow 
it to be as comprehensive as possible but close enough to the annual exercise so that its results will 
not be out of date. 


C. Prioritization Committee 


Key stages of this exercise are expert-driven. It is important to have sufficient and diverse expertise 
on which to draw. The 2016 methodology review recommended that participants be as geographically 
diverse as possible and, depending upon the diseases and pathogens to be considered, could include 
experts in: 


1. Microbiology of severe pathogens, including virology, bacteriology and mycology 

2. Clinical management of severe infections 

3. Epidemiology, in particular during health emergencies 

4. Public health policy, including emergency response 

5. Animal health, including veterinarians expert in zoonoses from both livestock and wildlife 

6. Other experts, including anthropologists, bioethicists, and other relevant social sciences, as 
well as experts from the defence or security sectors familiar with biological weapons 


To the extent possible, it is also desirable to include both disease specific experts (where it is known 
in advance a pathogen or disease will be included in the long list to be reviewed) as well as the 
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authors of important review articles on emerging infectious disease. This will help ensure the 
presence of both narrow and broad expertise. 


A core group of representative experts should be convened as a Prioritization Committee. This group 
will provide expert guidance when running the annual prioritization exercise, provide many of the 
participants for the annual prioritization meeting, and be expected to make themselves available to 
undertake an emergency prioritization exercise, should it become necessary as a results of a novel 
disease or pathogens (Part 2). It may be necessary to supplement the Prioritization Committee with 
additional expertise in some circumstances, for example disease specific knowledge when conducting 
an emergency prioritization exercise. 


D. Triaging possible diseases and pathogens 


As a first step, it is necessary to identify a sub-set of all possible diseases that might be assessed 
more systematically for potential prioritization. 


In other settings, similar processes have developed exhaustive, resource intensive processes to 
accomplish this goal. In the case of a prioritization process under the R&D Blueprint, and taking into 
account the regular nature of the review and the limited resources available, a streamlined process 
has been developed: 


1. As an initial input diseases or pathogens on the previous prioritized list should be fed into the 
annual review. 


2. Any additional diseases forwarded for re-consideration by the previous prioritization exercise, 
or any emergency prioritization exercise (conducted in accordance with Part 2 of this 
document) should also be fed into the annual review. 


3. Furthermore, approximately three weeks prior to the convening of the meeting for the annual 
prioritization exercise, the Prioritization Committee will be requested each to provide and 
additional 2-3 diseases or pathogens that might be included in the exercise. 


To help guide the experts in preparing these submissions, a number of identified disease scenarios 
were identified (Annex 2). Participants should be reminded that the R&D Blueprint focuses on 
diseases “where there are no, or insufficient, preventive, and curative solutions”. As a result, 
participants should set aside diseases, even those with epidemic potential, for which there already are 
major disease control initiatives, an extensive R&D pipeline, existing funding streams, and/or 
established regulatory pathways for improved interventions. Examples of such diseases identified 
during previous R&D Blueprint prioritization exercises include HIV/AIDS, tuberculosis, malaria, avian 
influenza causing severe human disease, antimicrobial resistance (as a generic category), 
smallpox/monkeypox, and dengue. 


Diseases and Pathogens identified during steps 1-3 should then be compiled into a long-list of 
diseases and pathogens to be considered during the annual review. This long list should only include 
around 10-15 diseases or pathogens. If more diseases have been fed into the process, two rounds of 
blinded Delphi technique can be used to reduce the size of list to be considered. 


The long-list fed into the annual review should be compiled in such a manner as to anonymise 
submissions. Suitable experts from within WHO might also be requested to input diseases or 
pathogens on to this list as it will be later reviewed by external experts. 


A draft long-list should be compiled a week to 10 days prior to the meeting. WHO should then identify 
one to two members of Prioritization Committee to introduce the disease during the annual 
prioritization exercise. One expert should ideally be a subject matter expert, able to make a short 
introduction (2-3 min) as to why the disease or pathogen might be relevant to this process. If 
logistically feasible and based upon the composition of the Prioritization Committee, second expert 
might also review available data and comment (2-3 mins) on its potential relevance to this exercise. 
This second expert might not be an expert in that particular disease or pathogen but be suitably 
informed as to be able to review its potential relevance in short order. The two experts should not 
work together on their comments and should ideally represent a range of views and opinions. Both 
should be encouraged to use the 8 prioritization criteria discussed in Section F when preparing their 
comments. 


E. Reviewing the Long-list of potential diseases and pathogens 


Early in the meeting for the annual prioritization exercise (Annex 3), participants should be given the 
opportunity to review and discuss the content of the long-list of disease and pathogens. This should 
be a general discussion as to the composition of the list and not an in-depth assessment of each of 
the diseases on it. The landscape analysis commissioned by WHO (Section B) should be introduced, 
ensuring all participants are familiar with technical developments pertinent to diseases and pathogens 
previously prioritized. Experts asked to prepared comments on specific diseases or pathogens should 
not make those comments at this point (unless needed as part of a Deliphi technique to reduce the 
size of the long list of diseases to be considered during the meeting). 


The discussions at this step should enable additional diseases or pathogens to be added to the list. 
Alternatively, it may also be desirable to remove diseases or pathogens from the long list. If 
necessary a Delphi technique may be used to reduce the size of the long list to ensure adequate time 
during the meeting can be dedicated to considering each of the diseases. Any such decisions to 
amend the long list, and the rationale behind them, will need to be carefully recorded. 


By the end of this step, it is desirable to have developed a consensus on the contents of the long list 
so that participants begin the exercise from a common position. 


F. Prioritization criteria 


The 2015 WHO Consultation for Prioritization of Pathogens identified nine prioritization criteria. These 
were revised and reordered during the 2016 methodology review. The current prioritization criteria 
are: 


1. Human transmission; 

2. Medical countermeasures; 

3. Severity or case fatality rate; 

4.(a)(joint) The human/animal interface; 
4.(b)(joint) Other factors; 

6. The public health context of the affected area; 
7. Potential societal impacts; and 

8. Evolutionary potential. 


There was recognition that not all of these prioritization criteria were equally important that some 
weighting of individual components was necessary (Section G). 


To further elaborate the prioritization criteria, a series of associated statements to explore different 
facets of the criteria were developed during the 2016 methodology review and updated in light of the 
use of the methodology during the 2017 annual review. 


1. Human transmission 

) There is evidence of human to human transmission 

) There is widespread human to human transmission 

) There is more than one route of human to human transmission 

) The disease frequently involves infectivity before the onset of symptoms 

) The pathogen is able to remain infectious for a prolonged period in an infected individual 
when convalescent or apparently recovered 

f) There is evidence of superspreading events 

g) The disease is likely to be amplified in a healthcare setting 


2. Medical Countermeasures: commercialised products or advanced candidates (such as 
those undergoing clinical trials) 
a) Diagnostics which are effective and suitable for use in the field are not available 
b) Diagnostics which are effective and suitable for use in a clinic or local healthcare setting are 
not available 
c) Effective Diagnostics are available but are only suitable for use in specialised facilities 
d) Effective vaccines (human or animal, as appropriate) and prophylactics do not exist 
) Effective vaccines (human or animal, as appropriate) and prophylactics which are suitable for 
use in resource limited settings do not exist 
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f) 
9) 


h) 


Effective drugs or therapies do not exist 

Effective drugs or therapies which are appropriate for use in resource limited settings do not 
exist 

The outbreak cannot be controlled by the application of common public health measures 
(such as contact tracing, Isolation of infected patients, social distancing, closure of public 
events, schooling, changes to cultural practices, e.g. burial rights, vector control, strict 
management of livestock movement) 


3. Severity or case fatality rate 


a) 
b) 


The disease causes high mortality 
The disease frequently causes high morbidity, including severe complications or sequelae 


4.(a) Human/Animal Interface 


The involvement of animals in transmitting (including arthropods) the disease to people is well 
characterized 

There are transmission routes from animals (including arthropods) to humans likely to result 
in high levels of human infections 

The pathogen is capable of infecting multiple animal species 

The animal species transmitting the disease are widely distributed 

The animal species transmitting the disease is abundant 

Arthropod(s) are responsible for transmitting the disease 

Arthropod(s) responsible for transmitting the disease are widely distributed 


4.(b) Other factors 


a) 
b) 


The geographic range of the pathogen has changed 

The pathogen shares relevant epidemiological and/or genotypic characteristics with agents 
which have caused important epidemics 

The natural disease does not result in robust protective immunity 

The disease carries a high risk of occupational exposure for those involved in a response 
(including for culling, vets, burial details, lab workers, first responders, healthcare workers) 
The pathogen is an agent likely to be used to cause deliberate outbreaks 


6. Public health context of the affected area 


a) 


b) 


The disease requires targeted surveillance (i.e. not likely to be detected by routine 
surveillance but which might be detected by active or sentinel surveillance) 

Disease control requires specialist interventions (such as highly skilled personnel; equipment, 
such as isolation units, respirators, PPE, etc.; and infection control measures) 


7. Potential societal impacts 


The disease has a disproportionate impact on special populations (such as pregnant women, 
children, immunocompromised, etc.) 

The disease can cause major social disruption 

The disease can cause major fear 

The disease can result in major economic impact 

The disease can result in a major disruption to healthcare delivery 


8. Evolutionary potential 


a) 
b) 
c) 


There is evidence of rapid pathogen evolution 
There is a trend towards increasing severity of the disease 
There is a trend towards the increasing transmissibilty of the pathogen 


It is not intended that participants in annual prioritization exercises review or amend the identified 
prioritization criteria or associated statements. This is a task reserved for a separate methodology 
review process. In accordance with established best practice, these two processes need to be clearly 
separated. 


G. Weighting prioritization criteria 


Each prioritization criteria may not have an equal impact on whether a disease needs to be prioritized. 
For example, its economic impact may be less important than its transmissibility. As a result, it is 
necessary to weigh the individual criteria. 


As an initial step, at the 2015 WHO Consultation for Prioritization of Pathogens the experts undertook 
an exercise to weigh the various identified criteria — ranking their relative impact on a public health 
emergency. However, a more quantitative approach was recommended. 


During the development of a more robust methodology in 2016, a second group of external group of 
experts was surveyed as to the relative importance of the prioritization criteria. Unlike the 2015 
assessment, a quantitative methodology was used. The overall 2016 assessment was: 


Human transmissibility : 32% 
Medical countermeasures: 21.90% 
Severity : 14.65% 
Human/animal interface : 9.42% 
Other contributing factors: 9.42% 
Public health context of the affected area: 6.13% 
Potential social impacts: 4.18% 
Evolutionary potential : 2.28% 


The results were similar to those obtained in 2015 but benefitted from the consequent conceptual 
development of the R&D Blueprint. This helped tailor the criteria to the specific needs of the Blueprint. 
For example, in this context the availability of medical countermeasures was felt to be of greater 
importance than when considering factors that affect a response to a public health emergence more 
broadly. As a result, this criterion was moved from being 4" in relative importance to oF 


It is not intended that participants in annual prioritization exercises review or amend the weightings of 
prioritization criteria. This is a task reserved for a separate methodology review process. In 
accordance with established best practice, these two processes need to be clearly separated. 


H. Assessing diseases and pathogens against the weighted prioritization 
criteria 


This task, and subsequent steps to review and discuss survey results will form the basis of most of 
the meeting convened for the annual exercise. 


In accordance with best practice, it is necessary to have separate weighting and assessment 
exercises. It is also important to use the components developed in earlier stages of the process 
(Sections F,G, I, and J) and to involve experts from different regions and disciplines (Section C). It is 
also necessary to provide all available evidence in support of the decisions being made. 


In general, it may be useful to schedule a series of working sessions to discuss the diseases and 
pathogens under review. This enables participants to share data not widely available, and to debate 
the diseases in the specific context of the R&D Blueprint. Strict time keeping will be important during 
this part of the annual prioritization exercise. A suitable chair will need to be identified in advance and 
they should be empowered to keep strictly to the allocated timings. 


If following the model agenda outline in Annex 3, No more than 15 minutes should be spent on each 
pathogen or disease. The two experts invited to make prepared comments should each be provided 
up to 3 minutes to make their remarks. The remaining time should be allocated for a discussion as to 
the relevance of the pathogen or disease to the prioritization exercise. Participants should be asked to 
focus on the prioritization criteria in Section F and to make all interventions exceedingly short. They 
should also be reminded that the R&D Blueprint focuses on diseases “where there are no, or 
insufficient, preventive, and curative solutions”. As a result, participants should set aside diseases, 
even those with epidemic potential, for which there already are major disease control initiatives, an 
extensive R&D pipeline, existing funding streams, and/or established regulatory pathways for 
improved interventions. 
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Following these discussions, participants should then be asked to complete the online survey. To 
simplify the scoring process, for each disease or pathogen participants are requested to agree or 
disagree with each of the statements detailed in Section F. Participants can select a relevant position 
on a sliding indicator which ranges from ‘Strongly Agree’ to ‘Strongly disagree’. Using the sliders 
facilitates the quantification of results, allowing answers to be coded into a specific score that can be 
adjusted by the identified weightings, and fed into an algorithm (section J) to determine an overall 
score for each disease or pathogen being assessed. 


Plenty of time should be set aside for this exercise and facilities provided for participants to work in a 
shared space. They should have the opportunity of interacting with each other individually or 
collectively to seek clarifications, discuss relevant data and take full advantage of the expertise 
present in the room. In general, it may be useful to involve all the participants in these discussions, 
sharing the insights generated as widely as possible. In some cases, participants might wish to be 
able to interact on a one-to-one basis. 


It might also be useful to use a rapporteur during these discussions (not one of the expert 
participants) as a summary of this discussion will provide important supporting evidence of how 
decisions where reached, in line with established best practice. 


It is not intended that participants in annual prioritization exercises review or amend the methodology 
being used to assess diseases and pathogens against the weighted prioritization criteria. This is a 
task reserved for a separate methodology review process. In accordance with established best 
practice, these two processes need to be clearly separated. 


lI. Data processing 


Answers provided by participants are collected through an online survey. The survey makes use of 
the prioritozation criteria and associated statements (Section F) and the associated weightings 
(Section G). 


To rank the short list of diseases, the participants complete an online survey tool. To simplify data 
collection, a visualization tool was built using the slide-bar function of R-Shiny. The tool makes use of 
a relative scale — comparing values between diseases rather than absolute values (i.e. the impact of 
scoring diseases A and B at 3 and 5 is the same as scoring the same diseases at 7 and 9). 
According to Saaty 1990: 
“Absolute measurement needs standard to make it possible to judge whether the alternative is 
acceptable or not. Absolute measurement is useful in areas where there is fairly good 
agreement on standards which are then used to rate alternatives on at a time’. 
This is not the case in the context of emerging and re-emerging infectious diseases where many 
characteristic remain unclear or unknown. 
Allowing participants, at a glance, to compare scoring between diseases helps to capture their views 
for each sub-criteria (statement) (Figure 3). The visualization tool also allows experts to record where 
a sub-criteria might be equally important for two diseases - setting the slider to the same value. It also 
intuitively allows the expert to represent the comparative ratio of importance of a criteria between 
diseases — by increasing or decreasing the gaps between their relative positions. The tool also 
allowed experts to opt out of scoring a particular disease or criteria - by setting the slider to zero. 


> Saaty, T. L. How to make a decision: the analytic hierarchy process. European journal of operational research 
48, 9-26 (1990) 


11 


2. There is widespread human to human transmission 


Ebola filovirus disease * 


Marburg filovirus disease * 


Rift Valley fever virus * 


FIGURE 3: Example of the scoring process. 


J. Prioritization algorithm 


To prioritize diseases a custom Analytic Hierarchy Process (AHP) is used. It is traditionally used to 
help policy makers in decision-making processes when confronted to complex decisions. The AHP 
method gives a complete and rational structure for the decision making process by representing the 
problem in hierarchic structure, evaluating and quantifying each criteria that influences the decision, 
linking them to the objectives, evaluating the alternatives and ranking them. 


AHP is useful for the prioritization of diseases because it can rank them according to prioritization 
criteria. To being with the problem is stratified. The algorithm used in this methodology has four 
levels: the first presents the aim of the process (prioritization of diseases); the second level covers the 
prioritization criteria; the third level presents the statements associated with the criteria; and the fourth 
level is the diseases being reviewed. 


The first step in the algorithm requires mapping the hierarchic structure of the prioritization problem. 
As an example, Figure 4 presents the first step of the prioritization algorithm for one of the criteria 
(human transmissibility). When fully implemented, similar hierarchical structures are mapped for each 
criterion. 


The second step of this algorithm weights the criteria, in accordance with Section G. Using the results 
of the 2016 methodology review, the criteria are classified according to their importance as follow: 


C,: human transmissibility; 

C2: the availability of countermeasures; 

Cz: severity (or case fatality rate); the potential scope of outbreak (risk of international spread); 
C41: the human animal interface 


ê Saaty, T.L. (2008). Decision making with the analytic hierarchy process. Int. J. Services Sciences, Vol. 1, No. 1, 2008. 
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C42: Other contribution factors; 

Cs: the public health context of the affected area; 
C7: potential societal impacts; 

Cg: the evolutionary potential. 


Prioritize the pathogens 


Figure 4: Hierarchic prioritization problem. 
Where C;,Co...C 8, SubC;...SubC7 and P4,P2...P10 represent the prioritization criteria, the sub-criteria for criterion 
Cı and the ten pathogens, respectively. 


The third step in the algorithm is to make a pairwise comparison between the pathogens for each sub- 
criterion (statement) by using the model developed for this methodology and being published in a 
forthcoming peer-reviewed article. The answers provided by the participating experts are used to 
make this comparison. The scores provided by the individual experts are averaged and the weighted 
scores computed. If there is inconsistency amongst the views held (and scores provided) by different 
experts when considered a specific statement, perhaps due to judgement differences, the 
discordance will be recorded and the group tasked with considering this facet of the criteria further. 


The final step of this algorithm is to compute the multi-criteria scores to rank the pathogens by using 
the results of steps 2 and 3. 


K. Compiling the list of priority diseases and pathogens 


WHO will present participants with a breakdown of the results of the survey and the output of the 
prioritization algorithm. The information provided should include details of the mean scores for each 
statement for each criterion as well as the deviation between participants’ answers. These results 
should be presented in an accessible format during a session specifically to review the results. 


In accordance with Section J, time will be needed to revisit discordant results, where there was 
particular disagreement in the views of participants. A working session might usefully be set aside to 
accomplish this (Annex 3). During this time, participants will have an opportunity to discuss the 
findings under each relevant prioritization criteria in more detail. This is effectively a second round of 
discussion, similar to that envisaged in Section H but focused specifically at those areas where there 
is greatest disagreement. 


Having considered each of the prioritization criteria (and associated statements) across all of the 
diseases being reviewed and revisited any particularly discordant results, a draft prioritized list should 
be made available to participants. This should include the comparative overall scores for each 
pathogen, if possible broken down with an indication of the comparative importance of each criterion 
in determining the overall score. 


As a final step, the draft prioritized list will need to be subjected to expert discussion to ensure the 
results are internally consistent and fit within the broader technical environment. Depending upon the 
views of participants, it may be necessary to adjust the output of the process. In general any alteration 
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to the ranking produced by this mechanism should be agreed by consensus and justifications 
recorded. Time for a substantive discussion will be required and a dedicated working session might 
be necessary (Annex 3). A final round of Delphi technique or anonymized voting may be required to 
ensure that the views of all participants are sufficiently reflected. 


Some consideration will also be required as to how best to present the results. For example, is there 
sufficient difference between the pathogens or diseases reviewed and adequate confidence in the 
results generated, to be able to produce a ranked listing? If not, would the clarity of a single list 
without any internal ranking be a sensible output, as was the case in the 2017 annual review. 
Alternatively, are the results better suited to produce a striated listing, where a number of pathogens 
or diseases are grouped together? For example, during the 2015 prioritization exercise diseases and 
pathogens were placed into two categories: those requiring urgent action; and serious, necessitating 
further action as soon as possible. Initially such a striation will likely need to be a qualitative process. 
As subsequent annual prioritization exercises are carried it, it may be able to identify quantifiable 
thresholds for such categories. 


L. Assessing confidence in the list 


It is important to assess the degree of confidence that might be placed in the prioritized list. The 
confidence is estimated by using the error propagation method. This method considers the variance in 
the original sample (the differences in the answers of the experts for the same criterions leads to 
different scores for that same criterion) and its propagation when manipulating the data for the 
prioritization. The variance in the original sample is systematically tracked through all the equations 
used for ranking in the AHP process which lead to a ranked list with discordance intervals. The 
detailed procedure developed specifically for use in this methodology in contained in the forthcoming 
peer-reviewed article. 


M. Closing the annual prioritization exercise 


At the end of the annual review, participants might also be asked to reflect on how the methodology 
might be further improved. Suggestions should be carefully recorded and fed into the next 
methodology review. 


Before closing the exercise, it is important to ensure that time is set aside for participants to reflect on 
the process and raise any issues they might have. Any identified issues should be openly discussed, 
potentially resulting in gathering more supporting evidence, caveats or other useful notes that may 
need to accompany the results of this exercise. 


N. Promoting the results of the exercise 


It is critical that both the prioritization process and the annual exercise are seen as transparent and 
robust. As much information should be made publicly available as possible. A comprehensive report 
of the annual prioritization exercise is a minimal requirement. 


It may also be desirable to promote the revised priority list of diseases or pathogens. This might 
usefully include press releases from WHO. It may also be desirable to explore opportunities to 
actively promote the list at appropriate public health events and venues. Peer reviewed publications 
might also be considered. 


PART 2 


Decision instrument for new diseases or pathogens, 
or one that is presenting in a new manner 


It is possible that an unusual outbreak involving a pathogen or disease might need to be prioritized 
between annual exercises. Such an event might involve a brand new pathogen, or a pathogen which 
has not been considered in past annual prioritization exercises (e.g. presenting in a modified or 
altered manner). This decision instrument is intended to guide users through considering available 
information and determining what action to take. It is not intended as formal guidance or binding in 
any manner but is intended as a tool to facilitate planning and response. 


The decision instrument helps the user navigate via a series of logical steps. The outcome of each 
step determines the next. Numerous different paths through the instrument are possible. It begins with 
WHO becoming aware of a novel public health risk relevant to the R&D Blueprint, progresses through 
internal and external interactions, and results in a range of different outcomes. In some cases, an 
outbreak might warrant convening an emergency review to grant the disease or pathogen preliminary 
priority status, thereby triggering a more comprehensive R&D response component. In other cases, 
the disease or pathogen, whilst having the potential to cause a public health emergency might not 
warrant immediate action (beyond monitoring ongoing events) and can be safely fed into the next 
annual prioritization exercise. There may times where there is insufficient information on which to 
base an assessment and contact with experts or additional consultations might be required. 
Alternatively, after careful reflection, no further action may be needed. 


The decision instrument is divided into two phases. The first is an internal assessment for WHO. It 
connects this process with the organization’s other efforts, including the annual prioritization process, 
disease or pathogen-specific programmes and consideration of the public health impact of an event. 
In general, should this there be evidence of a public health emergency, a more detailed assessment 
involving external expertise may be needed. If there is insufficient evidence, the user is guided to 
monitor the situation and gather more information until satisfied that no further action is necessary. 
This phase should start immediately on WHO becoming aware of a disease event and could last 
hours or days until external expertise is sought or a decision taken that no further action is necessary. 


The second phase takes the user through gathering relevant information from outside the WHO. It 
looks at properties of the pathogen involved, such as pathogenicity or transmissibility. The impacts of 
the disease are also considered, such as whether there is evidence of unusual patterns or significant 
disruption. The user is also guided to consider broader public health contexts, such as the availability 
of medical or public health countermeasures and, embracing the One Health approach, information 
on animal disease impact. These steps correspond to criteria used in the annual prioritization 
exercise. If a user identifies areas of concern in the pathogen’s properties, the disease’s impact or in 
the broader public health context, they are guided to initiate an emergency prioritization process. 
Such a process should take advantage of the Prioritization Committee convened under the Annual 
Prioritization Exercise (Part 1, Section C) and should take place within a week of the process being 
triggered. If an emergency prioritization exercise is not warranted, the user is guided to monitor the 
situation and gather more information until satisfied that no further action is necessary. This phase 
may last for days or weeks depending upon the nature of the disease event. 


If it is necessary to convene an emergency review, additional actions are envisaged. If a decision is 
taken that the disease or pathogen should be prioritized, a temporary working group on the R&D 
components of the disease should be convened. Such a group would be an important resource for 
WHO to map current capabilities. In particular, the temporary working group would assist in 
conducting a pipeline analysis — determining what possible countermeasures are in which stages of 
development. The pipeline analysis can then be fed into a research meeting which would bring 
together key stakeholders from different fields to consider identify sensible next steps. Ultimately, 
these resources can be combined to produce a series of relevant product profiles. If the review 
determines the disease is not an immediate priority, it is expected that it will be fed into the long list of 
diseases to be considered at the next annual review (Part 1, Section D). These additional activities 
phase might be pursued in the weeks or months following the emergency prioritization review. 


WHO becomes aware of a novel public health threat relevant to the R&D Blueprint 


Has the pathogen or disease (in current form) already been reviewed by a prioritization exercise? 
Y N 


Is there a substantial disease or 


i ? 
Has something changed? pathogen programme in WHO? 


Follow past decisions Report to the programme 


Gather disease & pathogen 
characterization data 


Is the disease highly transmissible? 


Are medical countermeasures “Don’t know” 


“No” ll 7 i i 
otoa unavailable or ineffective? to any 


Is the disease causing high mortality or 
frequently causing high morbidity? 


Is there a risk of spill over across the 
human/animal interface? 


No further Is the disease demonstrating unusual 
action patterns, evolving rapidly, or otherwise 
required poses a risk for other reasons? 


PROCESS INVOLVING EXTERNAL EXPERTS 


Are local 
resources likely 
to cope? 


Is the disease having a significant 
societal impact? 


For factors which may be relevant 
when considering these questions see 
Prioritization Criteria (Part 1 Section E) 


Convene a sub-group of the Prioritization Committee to run 
an emergency prioritization to determine provisional status 
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Re-run risk ranking exercise 


ANNEX 1 


Outline of the prioritization process 


Ensure Ensure multi- 
criteria fulfil disciplinary panel 
objectives (if using experts) 


Provide Ensure multi- 
definitions of disciplinary panel 
criteria & scores (if using experts) 


Provide Provide evidence 
definitions of to support 
criteria & scores decisions 


disease threats. Stockholm: ECDC; 2015. 


Use systematic 
methods & 
describe clearly 


Separate 
weighting & 
scoring exercises 


Ensure multi- Consider 
disciplinary panel methods for 
(if using experts) validating results 


Adapted from: European Centre for Disease Prevention and Control. Best practices in ranking emerging infectious 


http://ecdc.europa.eu/en/publications/Publications/emerging-infectious-disease-threats-best-practices-ranking.pdf 
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ANNEX 2 
Disease scenarios 


Disease scenarios involving pathogenicity 


° Diseases causing severe mortality 

° Diseases causing severe morbidity 

° Diseases with associated severe complications 
° Diseases with severe sequela 


Disease scenarios involving transmissibility 


Diseases with effective human-to-human airborne transmission 
Diseases with effective human-to-human sexual transmission 
Diseases with effective foodborne transmission 

Diseases with a common intermediate host 

Diseases with a common vector 

Diseases with a common reservoir 


Disease scenarios with unusual patterns 


Localised diseases beginning to spread 

° Diseases rapidly spreading 

e Diseases spreading to new areas 

° Diseases demonstrating novel resistance to common countermeasures 


Disease scenarios causing disruption 


° Diseases perceived by the general public to pose a particular risk 

° Diseases causing civil disruption 

° Diseases causing economic disruption 

Counter-measures 

° There are no, or insufficient effective counter measures 

° Counter-measures are too expensive, complicated or unavailable for widescale use 
° There is increasing resistance to relevant countermeasures 


Other disease scenarios 


e Diseases causing severe animal morbidity or mortality 
° Diseases resistant (or becoming resistant) to common countermeasures (including lack of 
vaccine or therapeutics) 


ANNEX 3 


Model Programme for Annual Prioritization Exercise Meeting 


9.00-9.30 


9.30-10.30 


10.30-11.00 


11.00-12.30 


12.30-13.30 


13.30-15.00 


15.00-15.30 


15.30-17.00 


09.00-10.30 


10.30-11.00 


DAY 1 


Opening session 


Welcome remarks 
Overview of meeting aims and prioritization methodology 


Session 1: Discussion of the long list of diseases and pathogens 
Introduction of landscape analysis 


(Participants will have the opportunity to discuss the content of the long list of 
diseases, explaining why they think items should be (or should not be) included. If 
necessary, the long list of diseases will be triaged to contain around 12 diseases.) 


Coffee Break 


Session 2: Discussion of diseases and pathogens to be prioritized 


(Invited experts will provide short introductions to the diseases and pathogens to be 
prioritized. Participants will have the opportunity to discuss each disease in turn. Both 
the introductions and participant discussion will make use of the prioritization criteria.) 


Lunch 


Session 3: Discussion of diseases and pathogens to be prioritized (Cont.) 


(Invited experts will provide short introductions to the diseases and pathogens to be 
prioritized. Participants will have the opportunity to discuss each disease in turn. Both 
the introductions and participant discussion will make use of the prioritization criteria.) 


Coffee break 


Session 4: Completion of the prioritization survey 
Overview of the survey 
Completion of the survey 


(Participants will be tasked with completing the survey — for each disease agreeing or 
disagreeing with statements derived from the prioritization criteria. Participants will 
have the opportunity to interact with each other to facilitate the completion of the 
surveys.) 


DAY 2 


Session 5: Presentation of survey results 


(WHO will present the results of the survey, highlighting where there is agreement 
and disagreement between participants on each element of the prioritization criteria. 
Participants will have the opportunity to compare results across the diseases and 
pathogens being prioritized.) 


Coffee break 
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11.00-12.30 


12.30-13.30 


13.30-15.00 


15.00- 15.30 


15.30-17.00 


Session 6: Discussion of discordant results 


(Participants will revisit those statements where there was particular variation in the 
views expressed, discuss associated data and have the opportunity to re-score 
them.) 


Lunch 


Session 7: Finalize ranking for the 2017 list of priority diseases 


(WHO will present the draft ranking of diseases and pathogens and in accordance 
with the validated methodology, participants will undertake a final round of the Deplhi 
approach to finalise the ranking.) 


Coffee Break 


Closing Session 


Review of exercise 

Opportunities to improve the process 
Plenary discussion (all participants) 
Closing remarks and next steps (WHO) 
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